Image clean-up and pre-coding

ABSTRACT

A method of filtering a digital image is described. A filter kernel is applied to a respective pixel in a set of pixels to smooth noise and preserve spatial frequencies associated with image edges in the digital image in accordance with a first filtering parameter. The filter kernel is a function of the respective pixel and has a closed form for the respective pixel. The filter kernel includes contributions from a first set of neighboring pixels and has a content-dependent normalization such that a sum of elements in the filter kernel equals a substantially fixed value.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional application60/579,840, entitled “Methods and Systems for Image Clean-Up andPre-coding,” filed on Jun. 14, 2004, the contents of which are herewithincorporated by reference.

FIELD OF THE INVENTION

The present invention relates generally to image processing, and morespecifically to image and video clean-up and pre-coding.

BACKGROUND

Conventional image noise removal (or reduction) algorithms may bedivided into two categories: statistical methods and kernel methods. Onestatistical method algorithm is median filtering. In median filtering, avalue for a respective pixel in a set of (noisy) pixels to be cleaned upin an image is determined as a median pixel value in a specified windowcentered at the respective pixel. While Median filtering may beeffective in removing or reducing impulse noise, it often has difficultyin removing Gaussian (white) noise and may blur the image (i.e., theimage may be smoothed). Blurring may be more pronounced when the windowis larger, for example, in images with a high percentage of impulsenoise.

Another statistical method, order-statistics (OS) filtering may offer areduced amount or degree of blurring. In OS filtering, the set of pixelsin the window are arranged as an ordered sequence and the respectivepixel is replaced by a linear combination of this sequence usingsuitable pre-determined weights. However, the same window (shape andsize) and the same weights may be used for each pixel in the image. As aconsequence, it is often difficult to preserve or maintain an overallimage sharpness.

Kernel methods, such as moving average (MA) filtering, infinite impulseresponse (IIR) or autoregressive moving average (ARMA) filtering (i.e.,MA in conjunction with autoregressive feedback), and convolutionfiltering, may be more effective in reducing Gaussian noise, but may beless effective in reducing impulse noise. In addition, depending onfilter coefficients (also referred to as filter weights), kernel methodsmay have even more difficulty than the statistical methods in preservingimage sharpness.

Conventional image filtering, including the statistical and the kernelmethods, often achieve noise reduction by image smoothing, and thus, bysacrificing image sharpness. This may lead to excessive blurring of theimage. While there have been attempts to modify these methods topreserve a certain amount of image sharpness (for example, through theuse of a convolution mask for each pixel in accordance with an inversegradient), such approaches entail increased computational cost andcomplexity, and often use multi-stage processing (i.e., numerousiterations of the image processing) of the image.

Recent advances in image noise removal include the “maximum aposteriori” (MAP) and variational approaches. The MAP approach isstatistical (i.e., discrete) in nature. The variational approach isanalytical and is often proposed as a minimization problem of an energyfunctional, which is often defined as a summation of a bending orinternal (“a prior”) energy functional. While the internal energyfunctional governs the output image quality, the external energyfunctional measures the proximity to the input image to be cleaned up. Apositive constant is used as a parameter for balancing image(smoothness/sharpness) quality and fidelity of the output “clean” imagein comparison with the input noisy image (governed by the externalenergy). The steepest decent approach to solving the Euler-Lagrangeequation of the energy minimization problem gives rise to the (isotropicand anisotropic) diffusion and diffusion-reaction partial differentialequations (PDE). While the variational approach and other recent relatedapproaches (such as numerical solutions of an anisotropic diffusion ordiffusion-reaction PDE) usually provide an improvement over theconventional algorithms discussed above, the improvement often entailsincreased computational cost and complexity, and often uses multi-stageprocessing of the image.

There is a need, therefore, for an improved image processing approachthat removes or reduces noise in an image while substantially preservingimage content (such as image texture and image edges). There is also aneed for reduced computational cost, reduced complexity and one-pass ora reduced number of stages in processing of images.

SUMMARY

A method of filtering a digital image is described. A filter kernel isapplied to a respective pixel in a set of pixels to smooth noise andpreserve spatial frequencies associated with image edges in the digitalimage in accordance with a first filtering parameter. The filter kernelis a function of the respective pixel and has a closed form for therespective pixel. The filter kernel includes contributions from a firstset of neighboring pixels and has a content-dependent normalization suchthat a sum of elements in the filter kernel equals a substantially fixedvalue. Alternately stated, the sum of elements equals or approximatelyequals a fixed value.

The applying may include a single pass for the respective pixel. Theapplying may be performed for each pixel in the set of pixels to producea filtered digital image. Applying the filter kernel to each pixel inthe set of pixels one or more additional times may leave the filtereddigital image substantially unchanged.

The first filtering parameter may determine a boundary between smoothingand sharpening. The first filtering parameter may substantially maintaintexture information in the digital image by reducing smoothing of noiseat a position in the digital image containing spatial frequenciesgreater than a threshold.

Non-central elements in the filter kernel may include magnitudes ofdifferences between pixels in the first set of neighboring pixels and arespective pixel in a discretization of an anisotropic diffusionequation. A central pixel in the filter kernel may correspond to therespective pixel. The first filtering parameter may map a time step inthe discretization of the anisotropic diffusion equation to acontent-dependent scale. A conductivity in the discretization of theanisotropic diffusion equation may be a function of a wavelettransformation. The discrete conductivity elements from the conductivityfunction may be implemented as a look-up table.

The filter kernel may correspond to a window of size 2 m+1 by 2n+1 andmay include the first set of neighboring pixels. m and n for the filterkernel may be a function of a pixel size. m may be equal to n.

In some embodiments, the method may further include modifying a color orcolor component of the pixel using a filter kernel.

In some embodiments, the filter kernel includes contributions from asecond set of neighboring pixels and is further in accordance with asecond filtering parameter.

In an alternate embodiment of a method of filtering a digital image, thefilter kernel is applied to the respective pixel in the set of pixels tosmooth noise and preserve spatial frequencies associated with imageedges in the digital image in accordance with the filtering parameter.The filtered pixel corresponds to

${\sum\limits_{m,n}{\frac{1}{\gamma}{M \otimes U}}},$where M is a closed-form array that is a function of the respectivepixel and has a window size of 2 m+1 by 2n+1 (m and n are positiveintegers), M includes contributions from a 2 m+1-by-2n+1 set ofneighboring pixels, U is a sub-array in the set of pixels and includesthe respective pixel and the set of neighboring pixels, {circle around(×)} indicates element-by-element multiplication of elements in M andelements in U, and γ is a time to content-dependent scale, resulting innormalization of the filter kernel M such that a sum of elements in

$\frac{1}{\gamma}M$equals or approximately equals a fixed value.

The method of, and related apparatus for, filtering the digital imageoffer reduced noise while substantially preserving image content. Themethod and related apparatus also offer reduced computational cost,reduced complexity and one-pass or a reduced number of stages inprocessing of the digital image.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, reference should be made tothe following detailed description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a flow diagram illustrating an embodiment of a process offiltering a digital image.

FIG. 2 is a flow diagram illustrating an embodiment of a process offiltering a digital image.

FIG. 3 is a block diagram illustrating an embodiment of a client or aserver.

FIG. 4 is a block diagram illustrating an embodiment of a device.

FIG. 5 is a flow diagram illustrating an embodiment of a process offiltering a digital image.

FIG. 6 is a block diagram illustrating a transition parameter.

FIG. 7 is a block diagram illustrating local coordinates at a point onan image edge.

FIG. 8 is a flow diagram illustrating an embodiment of a process ofdetermining a filter kernel.

FIG. 9 is a block diagram illustrating a set of pixels in an image tile.

FIG. 10 is a flow diagram illustrating an embodiment of a process ofconstructing a filter kernel to a respective pixel at the corner of animage tile.

FIG. 11 is a flow diagram illustrating an embodiment of a process ofconstructing a filter kernel to a respective pixel on the boundary of animage tile.

FIG. 12 is a flow diagram illustrating an embodiment of a process ofconstructing a filter kernel to a respective pixel in the interior of animage tile.

FIG. 13 is a flow diagram illustrating an embodiment of a process ofconstructing and using a filter kernel.

FIG. 14 is a flow diagram illustrating an embodiment of a process ofconstructing and using a filter kernel.

FIG. 15 is a flow diagram illustrating an embodiment of a process offiltering a digital image.

FIG. 16 is a block diagram illustrating an embodiment of a process offiltering a P or B frame for or during video compression.

FIG. 17 is a block diagram illustrating an embodiment of a process offiltering a macro-block for or during video compression.

FIG. 18 is a block diagram illustrating an embodiment of a process offiltering a three-dimensional digital image.

FIG. 19 is a block diagram illustrating embodiments of unfiltered andfiltered digital images.

Like reference numerals refer to corresponding parts throughout thedrawings.

DETAILED DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the present invention. However, it will beapparent to one of ordinary skill in the art that the present inventionmay be practiced without these specific details. In other instances,well-known methods, procedures, components, and circuits have not beendescribed in detail so as not to unnecessarily obscure aspects of theembodiments.

An image processing method, and related apparatus and systems, aredescribed. The image processing is centered on a computationallyefficient filter kernel that may improve image quality (for example,reduce or remove noise), reduce a communication bandwidth and/or reducestorage size of a digital image or a set of digital images (such asvideo) while preserving and/or enhancing image content. The filterkernel may be included as a pre-filter or pre-coder in an image or videocompression system that uses an image or video compression methodologysuch as JPEG, JPEG-2000, MPEG, H263, or H264. In video compressionapplications, the filter kernel may be used to pre-filter I frames, Pframes, B frames and/or macro-blocks, as well as in 3-dimensionalpre-filtering of so-called motion compensated image cubes. The filterkernel may be used in scanners, printers, digital cameras (in cellulartelephones, in other portable devices, and/or in stand-alone cameras)and camcorders, as well as in post-processing of images. The filterkernel may be implemented in software and/or hardware, such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP) or other integratedcircuit. In some embodiments, the filter kernel may have a closed form.The filter kernel may be applied in one-pass to a set of pixelscorresponding to a respective image to produce a filtered image.Subsequent applications of the filter kernel to the filtered image mayleave the filtered image substantially unchanged. For example, changesin a magnitude at one or more spatial frequencies may be less than 5 to10% of a pre-filtered value. If the filter kernel is applied more thanonce, one or more smoothness/sharpness parameter(s) associated with thefilter kernel may be unchanged or may be modified.

The filter kernel is non-linear and content-dependent (henceforthreferred to as a content-dependent filter or CDF). For a respective setof pixels in the image, a 2-dimensional CDF corresponds to a window orfilter size of 2 m+1 by 2n+1 (for arbitrary positive integers m and n).The CDF is based on an anisotropic diffusion model and/or an anisotropicdiffusion-reaction model. A gradient operation in a conductivity term inthe anisotropic diffusion model may be replaced by a wavelettransformation in the horizontal and/or vertical directions. Rather thancomputing a numerical solution to a resulting partial differentialequation (PDE), which requires numerous iterative steps, one or moretime-to-scale transformations are introduced. These transformations mapa constant time-step parameter in a discretization of the PDE to one ormore content-dependent scales that, in general, vary for each pixel inthe image or a subset of the image. The mapping is in accordance withdifferences between a central pixel in the window and respective pixelsin one or more sets of neighboring pixels in the window and/or by awavelet transformation. In general, therefore, the resulting CDF variesfor each pixel to be process, i.e., filter coefficients or filterweights vary for different pixel values (or, more generally, with localimage content). The CDF, however, may have a closed form. As such, theimage processing of each pixel may be performed in a single operation,i.e., the CDF may be a one-pass filter. In some embodiments, the CDF maybe implemented in several passes, but with fewer operations or stagesthan conventional image filters (which may use up to 100 operations orstages).

The time-to-scale transformations (also called time-to-scale mappings)during construction of the CDF may take different forms depending on adesired window size of the CDF. The window size may be in accordancewith a pixel size in the image and/or an image resolution. For example,for small pixel sizes (high resolution) noise in the image may extendover several pixels. As a consequence, a larger window may be used insuch circumstances. The description below of derivations for severalfilter window sizes (including a 3×3 window) may be extended toarbitrary window size, as well as to filters with 3-dimensional windowsfor removing or reducing image noise in 3-dimensional images. As such,the derivations are considered illustrative of a general approach. Thetime-to-scale transformations may include at least one parameter thatmay be used to adjust a smoothness/sharpness for the image. At least theone parameter may be pre-defined or may be adjusted, automaticallyand/or manually. The adjusting may be in accordance with lightingintensity, shutter speed and/or image content in the image. At least theone parameter may maintain texture information in the image by reducinga smoothing of noise at a position in the image containing spatialfrequencies greater than a threshold.

Attention is now directed towards embodiments of a process for filteringa digital image and the related apparatus and systems. FIG. 1 is a flowdiagram illustrating an embodiment 100 of a process of filtering adigital image. A window size for the filter kernel, including the firstset of neighboring pixels is determined (112) for the filtering process.A filter kernel, such as a filter kernel for a CDF, is determined orconstructed for a respective pixel in a first set of neighboring pixels(110). The filter kernel smoothes noise and preserves spatialfrequencies associated with image edges in accordance with a firstfiltering parameter. A conductivity associated with the filter kernel isdetermined using an optional wavelet transformation (114). Contributionsto the filter kernel from a second set of neighboring pixels may beoptionally determined in accordance with a second filtering parameter(116). The filter kernel is applied to each pixel (including the centerpixel) in the first set of neighboring pixels to produce a filtereddigital pixel (118). A color of the respective pixel is optionallymodified using a corresponding filter kernel (120). The process may berepeated one or more times (122) for additional pixels in the set ofpixels. The process may be repeated one or more times for one or morecolor channels. The respective color channels may be RGB, Y Cb Cr, YUV,or YIQ color channels. The process 100 may be repeated a plurality oftimes including at least one iteration for at least a subset of pixelsin the image. In some embodiment, multiple pixels may be processed inparallel, using parallel processors or circuits. The process 100 mayinclude fewer operations or additional operations. In addition, two ormore operations may be combined and/or the order of the operations maybe changed.

FIG. 2 is a flow diagram illustrating an embodiment of a process 200 offiltering a digital image. A client device 210 (e.g., a device thatincludes an image sensor) transmits a set of pixels (214) to a serverdevice (e.g., a device that includes a processor, DSP, or ASIC) 212. Theset of pixels are received (216). One or more filter kernels areoptionally determined or constructed for each pixel in the set of pixels(218). A respective filter kernel is applied to each pixel in the set ofpixels to produce a filtered digital image (220). The filtered digitalimage is transmitted (222) to the client device 210. The filtereddigital image is received (224). The process 200 may include feweroperations or additional operations. In addition, two or more operationsmay be combined and/or the order of the operations may be changed.

FIG. 3 is a block diagram illustrating an embodiment of a clientcomputer, client device or a server device (e.g., a device that includesa processor, DSP, or ASIC) 300. The client computer, client device orserver device 300 may include at least one data processor, videoprocessor and/or central processing unit (CPU) 310, one or more optionaluser interfaces 314, a communications or network interface 322 forcommunicating with other computers, servers and/or devices 400 (FIG. 4),memory 324 and one or more communication busses or signal lines 312 forcoupling these components to one another. The at least one dataprocessor, video processor and/or central processing unit (CPU) 310 maysupport multi-threaded and/or parallel processing. The user interface314 may have one or more displays 316, keyboards 318 and/or pointers 320(such as a mouse). The client or server device 300 may include one ormore cache memories 308, which may be used as a rolling cache for atleast a subset of the set of pixels.

Memory 324 may include high-speed random access memory and/ornon-volatile memory, including ROM, RAM, EPROM, EEPROM, one or moreflash disc drives, one or more optical disc drives and/or one or moremagnetic disk storage devices. Memory 324 may store an operating system326, such as LINUX, UNIX, OS9 or WINDOWS, that includes procedures (or aset of instructions) for handling basic system services and forperforming hardware dependent tasks. Memory 324 may also storecommunication procedures (or a set of instructions) in a networkcommunication module 328. The communication procedures are used forcommunicating with one or more computers, servers and/or devices 400(FIG. 4).

Memory 324 may also include the following elements, or a subset orsuperset of such elements, including image processing module 330 (or aset of instructions), wavelet transformation module 336 (or a set ofinstructions), color transformation module 338 (or a set ofinstructions), encoding/decoding module 340 (or a set of instructions),such as JPEG or MPEG, one or more sets of pixels 342 and/or one or morefiltered digital images 344. The image processing module 330 may includefilter kernel generator 332 and/or one or more filter kernels 334. Insome embodiments, memory 324 further includes an image compressionmodule 350, used to compress the filtered digital images 344, using animage or video compression methodology (such as JPEG, JPEG-2000, MPEG,H263, or H264) to produce filtered and compressed images 352.

Although FIG. 3 shows the client or server 300 as a number of discreteitems, FIG. 3 is intended more as a functional description of thevarious features which may be present in the client or server 300 ratherthan as a structural schematic of the embodiments described herein. Inpractice, and as recognized by those of ordinary skill in the art, thefunctions of the client or server 300 may be distributed over a largenumber of servers or clients, with various groups of the clients orservers performing particular subsets of those functions. Items shownseparately in FIG. 3 could be combined, some items could be separatedand/or additional items may be included.

FIG. 4 is a block diagram illustrating an embodiment of a device 400,such as a cell phone that includes a camera, a digital camera orcamcorder. The device 400 may include one or more data processors, videoprocessors and/or central processing units (CPU) 410, one or moreoptional user interfaces 414, one or more image sensors 418, acommunications or interface 422 (such as USB) for communicating withother computers, servers and/or devices 400, memory 424 and one or morecommunications busses or signal lines 412 for coupling these componentsto one another. The one or more data processors, video processors and/orcentral processing units (CPU) 410 may support multi-threaded and/orparallel data processing. The user interface 414 may have one or moredisplays 416 and/or one or more keys or key pads. The device 400 mayalso include a power supply 420. The device 400 may include one or morecache memories 408, which may be used as a rolling cache for at least asubset of the set of pixels.

Memory 424 may include high-speed random access memory and/ornon-volatile memory, including ROM, RAM, EPROM, EEPROM, one or moreflash disc drives, one or more optical disc drives and/or one or moremagnetic disk storage devices. Memory 424 may store an embeddedoperating system 426, such as LINUX, OS9, UNIX or WINDOWS or a real-timeoperating system (e.g., VxWorks by Wind River Systems, Inc.) suitablefor use on industrial or commercial devices. The embedded operatingsystem 426 may includes procedures (or a set of instructions) forhandling basic system services and for performing hardware dependenttasks. Memory 424 may also store communication procedures (or a set ofinstructions) in a communication module 428. The communicationprocedures are used for communicating with one or more computers,servers and/or devices 400.

Memory 424 may also include the following elements, or a subset orsuperset of such elements, including image processing module 430 (or aset of instructions), wavelet transformation module 436 (or a set ofinstructions), color transformation module 438 (or a set ofinstructions), encoding/decoding module 440 (or a set of instructions),such as JPEG or MPEG, one or more sets of pixels 442. The imageprocessing module 430 may include filter kernel generator 432 and/or oneor more filter kernels 434. The device 400 may also include one or morefiltered digital images 444. The filtered digital images 444 may bestored in the memory 424 or in a separate non-volatile memory, such asflash memory, which may be removable. In some embodiments, memory 424further includes an image compression module 450, used to compress thefiltered digital images 444, using an image or video compressionmethodology (such as JPEG, JPEG-2000, MPEG, H263, or H264) to producefiltered and compressed images 452.

Although FIG. 4 shows the device 400 as a number of discrete items, FIG.4 is intended more as a functional description of the various featureswhich may be present in the device 400 rather than as a structuralschematic of the embodiments described herein. Items shown separately inFIG. 4 could be combined, some items could be separated and/oradditional items may be included.

FIG. 5 is a flow diagram illustrating an embodiment 500 of a process offiltering a digital image or video. A noisy image is received (510). Theimage is optionally pre-processed (512), including analog-to-digitalconversion (if necessary) and/or a color transformation. One or morefilter kernels are applied to multiple channels (514). An image in arespective color space is cleaned up (516). The image is encoded (518),for example, using JPEG or MPEG compression. The flow diagram 500 mayinclude fewer operations or additional operations. In addition, two ormore operations may be combined and/or the order of the operations maybe changed.

Attention is now directed towards the filter kernel for the CDF. Asdiscussed previously, conventional image processing for noise removal orreduction include MAP and variational approaches. Approaches based onthe anisotropic diffusion PDE, as well as the anisotropicdiffusion-reaction PDE, are used for the variational approach. Here anenergy functional is minimized, so that the corresponding conductivityfunction c of the diffusion model is not constant, resulting in anon-linear PDE. In general, such non-linear PDEs are solved numericallyusing an iterative process.

Some conventional approaches extend the approaches based on theanisotropic diffusion PDE by defining the conductivity c as a function,c(p)=φ′(p)/p, corresponding to a differentiable non-decreasing functionφ(p) with φ(0)=0. (If φ(p) is an internal energy density function of aninternal energy functional, a steepest decent approach to solving theEuler-Lagrange equation in a minimum-energy problem results in theanisotropic diffusion PDE for a discrete formulation of the externalenergy and the diffusion-reaction PDE for a continuous formulation ofthe external energy). Function φ″(p) 612 (where the double quote mark ″denotes a second derivative) as a function of p 610 is illustrated inFIG. 6, with p=|∇u|, the magnitude of a spatial gradient of u, which isa (unknown) solution to the anisotropic diffusion equation at a timeinstant. One popular choice for the internal energy density function isφ(p)=p. For this choice the corresponding conductivity functionc(p)=1/p. Small values of p 610 correspond to fast diffusion of theimage content. As consequence, image details such as image texture andlight image edges may be lost during the image processing. Large valuesof p 610 correspond to sharpening of edges in the image content. Whilethere is a continuous variation between these extremes, a zero crossing,referred to as a transition parameter p_(o) 614, separates the tworegimes.

FIG. 7 is a block diagram illustrating local coordinates 700, includinga transverse unit vector t 710 and a normal unit vector n 712, at apoint on an image edge. When p=|∇u| (FIG. 6) is larger than thetransition parameter p_(o) 614, sharpening occurs due to backwarddiffusion along the normal unit vector n 712. Note that the normal unitvector n 712 is a unit vector in the direction of the gradient of the(unknown) solution u, and the transverse unit vector t 710 is orthogonalto n using the right-hand rule. When the p=|∇u| (FIG. 6) is smaller thanthe transition parameter p_(o) 614 (FIG. 6), minimal diffusion occursalong the normal unit vector n 712 and hence blurring is less visible.For a suitable choice of the transition parameter p_(o) 614, imagesharpness may be maintained or enhanced while image noise is removed orreduced. As discussed previously, however, these and other models basedon the anisotropic diffusion PDE have increased complexity and anincreased amount of computation used to find numerical solutions.Typically, these and other solutions of the non-linear anisotropicdiffusion PDE use numerous iterations or multi-stage image processing.For example, the PDE may be discretized using a constant time-stepparameter Δt to formulate non-linear difference equations. A noisy inputimage is used as an initial condition of the PDE. Numerical solutions,corresponding to a “clean” output image, typically involve numerousiterative steps or operations (up to 100 or more) and also depend on anad hoc choice of a stop-time for terminating the iterative process.

In the derivation of a CDF described below, the anisotropic diffusionPDE is discretized. In other embodiments, related non-linear PDEs, suchas the diffusion-reaction PDE, may be discretized. The noisy image isused as the initial condition. The conductivity function c(p) is definedas a function of p=|∇u|. In exemplary embodiments, the conductivityfunction may be

$\begin{matrix}{{{c(p)} = {\mathbb{e}}^{\frac{- p^{2}}{2p_{o}^{2}}}},} \\{{{c(p)} = \frac{p_{o}^{2}}{p_{o}^{2} + p^{2}}},} \\{or} \\{{c(p)} = {{\min\left( {1,\frac{1}{p}} \right)}.}}\end{matrix}$

Instead of solving the PDE numerically, the time parameter is mapped toa pixel-dependent scale. This scale may eliminates the use of thestop-time and may allow image processing to be implemented in one-pass.As an illustrative embodiment, a 3×3 filter window is considered.Results for 3×5, 5×5 and 7×7 filter windows are also shown. As will beclear to those skilled in the art, CDFs with larger filter sizes, aswell as 3-dimensional CDFs, may also be constructed.

The CDF, derived below, has a finite filter length in both horizontaland vertical directions (3-dimensional embodiments of the CDF also havea finite length in a third direction). Each pixel in at least a subsetof the image that is filtered may have its own CDF filter. In someembodiments, theses CDF filters correspond to a window having asymmetric shape in the horizontal and vertical directions. The windowmay correspond to 2 m+1 by 2n+1 pixels, where m and n are arbitrarypositive integers (for 3-dimensional embodiments, a dimension in thethird direction is q). In an exemplary embodiment m and n are eachpositive integers between 1 and 50 (and for 3-dimensional embodiments, qmay be a positive integer between 1 and 10). In some embodiments, m maybe equal to n, i.e., the window may be square. In some embodiments, thewindow may rectangular. In some embodiments, the window may have anasymmetric shape, such as an even-by-odd number of pixels or anodd-by-even number of pixels. The CDF incorporates several additionalfeatures than in the formalism described above. In the process, theimage processing may be implemented in a single pass, i.e., in anon-iterative fashion.

In order for the CDF to be more effective, in some embodiments entriesin the window may not have arbitrary zero filter coefficients orweights. For example, when designing a 3×3 filter, corner entries in afilter matrix F_(ij) ^(k) are determined according to the conductivityfunctions of the anisotropic diffusion model. Thus, in some embodimentsall of geographic directions with respect to the center pixel in thewindow are discretized in accordance with the size of the window,thereby determining entries for all the elements in the window.

The use of a conductivity function c(|∇u|) that is a function of themagnitude of the spatial gradient of the unknown solution u may posechallenges. In particular, since the unknown solution u is used toinitially represent the input image data, it may be contaminated withnoise and the spatial gradient values of u may be inexact and mayexhibit strong fluctuations. To address this, the spatial gradient ofthe input image in the discrete conductivity functions may be replacedby a wavelet transform of the input image in the horizontal and/orvertical directions. In an alternative embodiment, the spatial gradientof the input image in the conductivity functions may be modified suchthat a Gaussian convolution is performed on the (unknown) solution ubefore the gradient operation is applied. While the CDF may be appliedin one-pass, if the CDF is applied more than once subsequent iterationsmay or may not use the modified version of the conductivity functionc(|∇u|) described above, since the filtered image may no longer benoisy.

In addition, time-to-scale transformations (also referred to astime-to-scale mappings) are introduced. These transformations map theconstant time-step parameter Δt (henceforth denoted by λ) in thediscretization of the anisotropic diffusion PDE to one or morepixel-dependent scales. These scales are formulated as functions of thecenter pixel in the input noisy image, as well as its neighboring pixelsin the filter window. As described further below, the time-to-scaletransformations are associated with one or more parameters thatdetermine an image smoothness/sharpness adjustment. The one or moreparameters may be used to produce a pleasing blur effect while keepingthe sharpness of certain distinct image edges substantially intact. Theuse of the time-to-scale transformation allows construction of thefilter kernel for the CDF without numerically solving the anisotropicdiffusion or other PDEs. As a consequence, when applying the CDF toremove or reduce noise from an image, the iterative steps and astop-time k may not be needed.

After discretization, the anisotropic diffusion PDE may be expressed as

${u_{ij}^{k + 1} = {u_{ij}^{k} + {{\frac{\Delta\; t}{h^{2}}\left\lbrack {C_{E}^{k}{\partial_{E}{+ C_{S}^{k}}}{\partial_{S}{+ C_{W}^{k}}}{\partial_{W}{+ C_{N}^{k}}}\partial_{N}} \right\rbrack}\; u_{ij}^{k}}}},$where u_(ij) ^(o) denotes the (i,j)^(th) pixel of the input image, andu_(ij) ^(k) denotes the solution (at the (i,j)^(th) pixel location) ofthe difference equation after k iterations (or k iterative time steps),with a constant time-step parameter denoted by Δt and adjacent pixeldistance denoted by h. C_(E) ^(k), C_(S) ^(k), C_(W) ^(k), and C_(N)^(k) denote discretization of the conductivity function c(|∇u|) alongEast (E), South(S), West (W) and North (N) geographic directions ororientations associated with a first set of neighboring pixels.Henceforth, these are referred to as discrete conductivity functions.Since the conductivity function c(|∇u|) is a function of the gradient ofthe unknown solution u=u(x,y,t), with spatial coordinates (x,y) and timevariable t, the difference equation is also dependent on the unknownsolution and is, therefore, image content-dependent. In the aboveformulation, ∂_(E), ∂_(S), ∂_(W), and ∂_(N) denote first-order spatialdifferences applied to the pixels. In particular,∂_(E) u _(ij) ^(k) =u _(i+1,j) ^(k) −u _(ij) ^(k)∂_(S) u _(ij) ^(k) =u _(i,j−1) ^(k) −u _(ij) ^(k)∂_(W) u _(ij) ^(k) =u _(i−1,j) ^(k) −u _(ij) ^(k)∂_(N) u _(ij) ^(k) =u _(i,j+1) ^(k) −u _(ij) ^(k).The difference equation may be re-expressed as

$u_{ij}^{k + 1} = {{\left\lbrack {1 - {\frac{\Delta\; t}{h^{2}}\left( {C_{E}^{k} + C_{S}^{k} + C_{W}^{k} + C_{N}^{k}} \right)}} \right\rbrack\mspace{11mu} u_{ij}^{k}} + {\frac{\Delta\; t}{h^{2}}C_{E}^{k}u_{i + {1j}}^{k}} + {\frac{\Delta\; t}{h^{2}}C_{S}^{k}u_{{ij} - 1}^{k}} + {\frac{\Delta\; t}{h^{2}}C_{W}^{k}u_{i - {1j}}^{k}} + {\frac{\Delta\; t}{h^{2}}C_{N}^{k}{u_{{ij} + 1}^{k}.}}}$This difference equation may, in turn, be re-formulated as an iterativepixel-dependent filtering process, with the filters represented bymatrix templates,

${F_{ij}^{k} = \begin{bmatrix}0 & {\Delta\; t\;{c_{N}^{k}/h^{2}}} & 0 \\{\Delta\; t\;{c_{W}^{k}/h^{2}}} & {1 - {\Delta\;{{t\left( {c_{N}^{k} + c_{E}^{k} + c_{S}^{k} + c_{W}^{k}} \right)}/h^{2}}}} & {\Delta\; t\;{c_{E}^{k}/h^{2}}} \\0 & {\Delta\; t\;{c_{S}^{k}/h^{2}}} & 0\end{bmatrix}},$where the discrete conductivity functions C_(E) ^(k), C_(S) ^(k), C_(W)^(k), and C_(N) ^(k) are functions of the first set of neighboringpixels as well as a center pixel, i.e.,u _(ij) ^(k) , u _(i+1,j) ^(k) , u _(i,j−1) ^(k) , u _(i−1,j) ^(k) , u_(i,j+1) ^(k) .C_(E) ^(k), C_(S) ^(k), C_(W) ^(k), C_(N) ^(k) are, therefore, alsodependent on the pixel locations (i,j). For notational simplicity,however, the subscripts i,j are not shown. In existing filteringapproaches that are iterative, the iterative filtering process (k=0, 1,2, . . . ) may be described by component-wise matrix multiplication(also called entry-wise multiplication or element-by-elementmultiplication) followed by summation of all resulting matrix componentsor

u_(i, j)^(k + 1) = ∑F_(ij)^(k) ⊗ u_(ij)^(k),where u_(ij) ^(k) is the central pixel in the set of pixelscorresponding to the filter window after k iterations.

To construct the filter kernel for a CDF with the 3×3 filter window, asecond set of neighboring pixels in the North East (NE), South East(SE), South West (SW) and North West (NW) directions are alsodiscretized. The difference equation may be extended to include these 4additional pixels,

$u_{ij}^{k + 1} = {u_{ij}^{k} + {{\frac{\Delta\; t}{h^{2}}\left\lbrack {\left( {{C_{E}^{k}{\partial_{E}{+ \cdots}}} + {C_{N}^{k}\partial_{N}}} \right) + {v\left( {{C_{NE}^{k}{\partial_{NE}{+ \cdots}}} + {C_{NW}^{k}\partial_{NW}}} \right)}} \right\rbrack}\; u_{ij}^{k}}}$where ν is a parameter that may be adjusted independently or that may becoupled to λ=Δt such that only one free parameter is used for imagesmoothness/sharpness adjustment. The difference equation may be changedin the CDF by defining a content-dependent scale value

${k_{ij} = {1 + {\frac{\alpha}{h^{2}}\left\lbrack {\left( {C_{E}^{0} + \cdots + C_{N}^{0}} \right) + {\nu\left( {C_{NE}^{0} + \cdots + C_{NW}^{0}} \right)}} \right\rbrack}}},$where α is a parameter corresponding to the image smoothness/sharpnessadjustment, and by introducing the time-to-scale transformation

${{\Delta\; t} = \left. \lambda\rightarrow\frac{\alpha}{k_{ij}} \right.},$which maps the constant time-step parameter Δt to the pixel-dependent,and thus image content-dependent, scale

$\frac{\alpha}{k_{ij}}.$The resulting CDF for cleaning up the noisy input image isu _(ij) =ΣF _(ij) {circle around (×)}u _(ij) ^(o).The notation {circle around (×)} stands for component-wise matrixmultiplication, and the notation Σ indicates summation over allcomponents or entries of the resulting matrix. Note that the iterationindex k is no longer present. The 3×3 CDF may be represented by a matrixtemplate

${F_{ij} = {\frac{1}{k_{ij}}\begin{bmatrix}\frac{\mu\; C_{NW}^{0}}{h^{2}} & \frac{\alpha\; C_{N}^{0}}{h^{2}} & \frac{\mu\; C_{NE}^{0}}{h^{2}} \\\frac{\alpha\; C_{W}^{0}}{h^{2}} & 1 & \frac{\alpha\; C_{E}^{0}}{h^{2}} \\\frac{\mu\; C_{SW}^{0}}{h^{2}} & \frac{\alpha\; C_{S}^{0}}{h^{2}} & \frac{\mu\; C_{SE}^{0}}{h^{2}}\end{bmatrix}}},$where another image smoothing/sharpening parameter μ=αν has beenintroduced (as mentioned previously, the parameter α may be coupled tothe parameter μ resulting in a single free parameter for imagesmoothing/sharpening adjustment.) In an exemplary embodiment, μ=α/4. TheCDF may also be written out explicitly as

$u_{ij} = {\frac{1}{k_{ij}}{\left\{ {\left( {{\frac{\mu\; C_{NW}^{0}}{h^{2}}u_{{i - 1},{f + 1}}^{0}} + {\frac{\alpha\; C_{N}^{0}}{h^{2}}u_{i,{j + 1}}^{0}} + {\frac{\mu\; C_{NE}^{0}}{h^{2}}u_{{i + 1},{j + 1}}^{0}}} \right) + \left( {{\frac{\alpha\; C_{W}^{0}}{h^{2}}u_{{i - 1},j}^{0}} + u_{ij}^{0} + {\frac{\alpha\; C_{E}^{0}}{h^{2}}u_{{i + 1},j}^{0}}} \right) + \left( {{\frac{\mu\; C_{SW}^{0}}{h^{2}}u_{{i - 1},{j - 1}}^{0}} + {\frac{\alpha\; C_{S}^{0}}{h^{2}}u_{i,{j - 1}}^{0}} + {\frac{\mu\; C_{SE}^{0}}{h^{2}}u_{{i + 1},{j - 1}}^{0}}} \right)} \right\}.}}$Note that the filter weights are a product of a magnitude of adifference between an adjacent pixel and the center pixel and aparameter (α or μ), corresponding to the first set of neighboring pixelsor the second set of neighboring pixels. Also note that thecontent-dependent scale value k_(ij) is relative to the center pixel andensures that the matrix F_(ij) is normalized, such that a sum of thefilter weights or elements is equal or approximately equal to a fixedvalue. The sum of the filter weights may vary slightly from the fixedvalue due to numerical processing artifacts and the like. Thenon-central elements in the CDF may determine smoothing of the inputimage.

Similarly, CDFs for 3×5, 5×5 and 7×7 windows may be determined orconstructed. For the 3×5 window, the CDF may be formulated as

λc_(i+1,j−2)/k_(i,j) βc_(i+1,j−1)/k_(i,j) αc_(i+1,j)/k_(i,j)βc_(i+1,j+1)/k_(i,j) λc_(i+1,j+2)/k_(i,j) ωc_(i,j−2)/k_(i,j)αc_(i,j−1)/k_(i,j) 1/k_(ij) αc_(i,j+1)/k_(i,j) ωc_(i,j+2)/k_(i,j)λc_(i−1,j−2)/k_(i,j) βc_(i−1,j−1)/k_(i,j) αc_(i−1,j)/k_(i,j)βc_(i−1,j+1)/k_(i,j) λc_(i−1,j+2)/k_(i,j.)Note that additional directions (NE-E, E-E, SE-E, SW-W, W-W, NW-W) havebeen taken into consideration by using the indices (i+2,j+1) . . .(i−2,j+1), and additional smoothing/sharpening parameters correspondingto additional sets of neighboring pixels have been included. The fourparameters (α, β, λ, ω) used in this formulation of the CDF may be tiedtogether to yield one or two independent parameters. Typical ratiosinclude 1/2, 1/4, 3/4, 3/8, 5/8, 7/8, 1/16, 3/16, 5/16, 7/16, 9/16,11/16, 13/16, 15/16, 1/32, 3/32, 5/32, 7/32, 9/32, 11/32, 13/32, 15/32,17/32, 19/32, 21/32, 23/32, 25/32, 27/32, 29/32, and 31/32. For the 5×5window, the CDF may be formulated as

γc_(i−2,j+2)/k_(i,j) λc_(i−1,j+2)/k_(i,j) νc_(i,j+2)/k_(i,j)λc_(i+1,j+2)/k_(i,j) γc_(i+2,j+2)/k_(i,j) λc_(i−2,j+1)/k_(i,j)βc_(i−1,j+1)/k_(i,j) αc_(i,j+1)/k_(i,j) βc_(i+1,j+1)/k_(i,j)λc_(i+2,j+1)/k_(i,j) νc_(i−2,j)/k_(i,j) αc_(i−1,j)/k_(i,j) 1/k_(i,j)αc_(i+1,j)/k_(i,j) νc_(i+2,j)/k_(i,j) λc_(i−2,j−1)/k_(i,j)βc_(i−1,j−1)/k_(i,j) αc_(i,j−1)/k_(i,j) βc_(i+1,j−1)/k_(i,j)λc_(i+2,j−1)/k_(i,j) γc_(i−2,j−2)/k_(i,j) λc_(i−1,j−2)/k_(i,j)νc_(i,j−2)/k_(i,j) λc_(i+1,j−2)/k_(i,j) γc_(i+2,j−2)/k_(i,j)The five parameters (α, β, λ, ω, ν) used in this formulation of the CDFmay be tied together to yield one or two independent parameters. Typicalratios include 1/2, 1/4, 3/4, 3/8, 5/8, 7/8, 1/16, 3/16, 5/16, 7/16,9/16, 11/16, 13/16, 15/16, 1/32, 3/32, 5/32, 7/32, 9/32, 11/32, 13/32,15/32, 17/32, 19/32, 21/32, 23/32, 25/32, 27/32, 29/32, and 31/32. Forthe 7×7 window, the CDF may be formulated as

σc_(i−3,j+3)/k_(i,j) χc_(i−2,j+3)/k_(i,j) δc_(i−1,j+3)/k_(i,j)κc_(i,j+3)/ δc_(i+1,j+3)/ χc_(i+2,j+3)/ σc_(i+3,j+3)/ k_(i,j) k_(i,j)k_(i,j) k_(i,j) χc_(i−3,j+2)/k_(i,j) γc_(i−2,j+2)/k_(i,j)λc_(i−1,j+2)/k_(i,j) νc_(i,j+2)/ λc_(i+1,j+2)/ γc_(i+2,j+2)/χc_(i+3,j+2)/ k_(i,j) k_(i,j) k_(i,j) k_(i,j) δc_(i−3,j+1)/k_(i,j)λc_(i−2,j+1)/k_(i,j) βc_(i−1,j+1)/k_(i,j) αc_(i,j+1)/ βc_(i+1,j+1)/λc_(i+2,j+1)/ δc_(i+3,j+1)/ k_(i,j) k_(i,j) k_(i,j) k_(i,j)κc_(i−3,j)/k_(i,j) νc_(i−2,j)/k_(i,j) αc_(i−1,j)/k_(i,j) 1/k_(i,j)αc_(i+1,j)/ νc_(i+2,j)/ κc_(i+3,j)/ k_(i,j) k_(i,j) k_(i,j)δc_(i−3,j−1)/k_(i,j) λc_(i−2,j−1)/k_(i,j) βc_(i−1,j−1)/k_(i,j)αc_(i,j−1)/ βc_(i+1,j−1)/ λc_(i+2,j−1)/ δc_(i+3,j−1)/ k_(i,j) k_(i,j)k_(i,j) k_(i,j) χc_(i−3,j−2)/k_(i,j) γc_(i−2,j−2)/k_(i,j)λc_(i−1,j−2)/k_(i,j) νc_(i,j−2)/ λc_(i+1,j−2)/ γc_(i+2,j−2)/χc_(i+3,j−2)/ k_(i,j) k_(i,j) k_(i,j) k_(i,j) σc_(i−3,j−3)/k_(i,j)χc_(i−2,j−3)/k_(i,j) δc_(i−1,j−3)/k_(i,j) κc_(i,j−3)/ δc_(i+1,j−3)/χc_(i+2,j−3)/ σc_(i+3,j−3)/ k_(i,j) k_(i,j) k_(i,j) k_(i,j)The nine parameters (α, β, λ, ω, ν, κ, δ, χ, γ) used in this formulationof the CDF may be tied together to yield one or two independentparameters. Typical ratios include 1/2, 1/4, 3/4, 3/8, 5/8, 7/8, 1/16,3/16, 5/16, 7/16, 9/16, 11/16, 13/16, 15/16, 1/32, 3/32, 5/32, 7/32,9/32, 11/32, 13/32, 15/32, 17/32, 19/32, 21/32, 23/32, 25/32, 27/32,29/32, 31/32, 1/64, 3/64, 5/64, 7/64, 9/64, 11/64, 13/64, 15/64, 17/64,19/64, 21/64, 23/64, 25/64, 27/64, 29/64, 31/64, 33/64, 35/64, 37/64,39/64, 41/64, 43/64, 45/64, 47/64, 49/64, 51/64, 53/64, 55/64, 57/64,59/64, 61/64, 63/64, and 65/64.

Attention is now directed towards embodiments of the wavelettransformation. FIG. 8 is a flow diagram illustrating an embodiment 800of a process of determining or constructing a filter kernel includingthe wavelet transformation described previously. An image u is received(810). A filtered image (L*u) is generated using a convolution of u witha low-pass filter L (812). In some embodiments, a weight sum of L in ahorizontal direction and/or a vertical direction may have a finitevalue. A difference operation of (L*u) is a wavelet transform of u inthe horizontal or vertical direction. If this wavelet transform is notapplied in one of the two directions, then a first order differenceoperation may be used in this direction instead. In some embodiments,other wavelet transforms may be used. A magnitude of a wavelettransformation (|∂(L*u)|), or a first-order difference, is determined(814). This value may be quantized to a respective conductivity level I(816). The wavelet transform (instead of the first-order differenceoperation) may be used with noisy input images. The results are mappedto a corresponding conductivity c(i*2^(m−n)) (818). The flow diagram 500may include fewer operations or additional operations. In addition, twoor more operations may be combined and/or the order of the operationsmay be changed.

As illustrated in FIG. 8, a noisy input image may be modified to yielddiscrete conductivity elements associated with the conductivityfunctions. In an exemplary embodiment, in place of the first orderdifference for the discrete gradient operation, the wavelettransformation may be implemented instead, for example, by applying adifference operation of a low-pass filtering of u in conjunction with alevel/conductivity chart. In an exemplary embodiment, the spatialgradient of u in c(|∇u|) may be replaced by some discrete spline-wavelettransform of u. For example, ∇u in c(|∇u|) may be replaced by ∂(L*u)=≡Wu, where L is a low-pass filter associated with some desirable cardinalB-spline, the notation * stands for the standard convolution operation,and W u denotes a discrete wavelet transform operation. The operators Land W may be determined in respective (geographic) directions withrespect to the center pixel of the noisy input image u. For example,W _(E) u _(i,j)=∂_(E)(L*u)_(i,j) , W _(S) u _(i,j)=∂_(S)(L*u)_(i,j), W_(W) u _(i,j)=∂_(W)(L*u)_(i,j) , W _(N) u _(i,j)=∂_(N)(L*u)_(i,j) , W_(NE) u _(i,j)=∂_(NE)(L*u)_(i,j) , W _(SE) u _(i,j)=∂_(SE)(L*u)_(i,j) ,W _(SW) u _(i,j)=∂_(SW)(L*u)_(i,j) , W _(NW) u _(i,j)=∂_(NW)(L*u)_(i,j)In alternative embodiments, a Gaussian convolution may be used as thelow-pass filter of u before the difference operation is applied to yieldthe desired discretization.

Table 1 illustrates an embodiment of a conductivity level/conductivityfunction chart, which associates the conductivity levels with values ofthe discrete conductivity elements. If an input image has r-bitgray-level for each channel (typically, r=8) and the conductivity levelis quantized to s bits (s is less than or equal to r and typicallyranges from 1 to 8), then the range of p=|∇u| is the sequence [0, . . ., 2^(r)−1]. If this sequence is quantized to 2^(r−s)[0, . . . ,2^(s)−1], then the conductivity function c(p) maps [0, . . . , 2^(s)−1]to [c(0), c(2^(r−s)), . . . , c(2^(r−s)(2^(s)−1))]. As discussedpreviously, typical examples of the conductivity functions includec(p)=e^(−p/K), c(p)=e^(−p) ² ^(/K) ² , and c(p)=K²/(K²+p²), where K is apositive constant for all image pixels, or may be adapted according toneighboring pixel values (for adjustment of image smoothness/sharpness).In an exemplary embodiment of K as a global scalar, K may be 50. Formany images, this number is larger than nearly 90% of the magnitudes ofthe gradient values in an image. The resulting chart may be used in alook-up procedure, as described further below with reference to FIGS. 13and 14, where a noisy 1-bit image (with a single image edge separatingthe 0-bit “black” portion and the 1-bit “white” portion) is consideredas input image, and the respective (center) pixel is a pixel on theimage edge (FIG. 13) and not on the image edge (FIG. 14).

TABLE 1 A conductivity level/conductivity function chart. Level (i) 0 1. . . 2^(s) − 1 conductivity c(0) c(2^(r−s)) . . . c(2^(r−s)(2^(s) − 1))c(i * 2^(r−s))

Attention is now directed towards additional embodiments that illustratethe determination or construction of the CDF and its use. FIG. 9 is ablock diagram illustrating a set of pixels 900 in an image, image tile,or image block, such as a macro-block of a P or B frame (in videocompression) of size 5×5. FIGS. 10, 11 and 12 illustrate theconstruction of corresponding 2-dimensional CDFs with up to a 3×3 windowfor sets of pixels labeled A, B and C, respectively, for removing orreducing noise in the 5×5 image tile. The set of pixels A in FIG. 9illustrates the construction and application of a CDF for the cornerpixel u₀₀ in FIG. 10. The set of pixels B in FIG. 9 illustrates theconstruction and application of a CDF for the boundary pixel u_(i0) inFIG. 11. The set of pixels C in FIG. 9 illustrates the construction andapplication of a CDF for an interior center pixel u_(ij) in FIG. 12. Forthe set of pixels labeled A, the center pixel is a corner pixel in theset of pixels 900. Hence, it will be replaced with a value based onpixel values of 3 of its neighbors. For the set of pixels labeled B, thecenter pixel is a boundary pixel in the set of pixels 900 and will bereplaced with a value based on pixel values of 5 of its neighbors. Forthe set of pixels labeled C, the center pixel is an interior pixel inthe set of pixels 900 and will be replaced with a value based on pixelvalues of 8 of its neighbors.

FIG. 10 is a flow diagram illustrating an embodiment 1000 of a processof constructing a filter kernel for a CDF and applying the CDF to acorner pixel u₀₀. The first order difference operation is applied to aset of pixels 1010 to produce elements 1012 that are mapped using alevel and conductivity chart (such as that shown in Table 1) to producediscrete conductivity elements 1014. Filter kernel 1016 is constructedby multiplying predefined subsets of the discrete conductivity elements1014 with the smoothing and sharpening parameters α and μ. In anexemplary embodiments, a is approximately between 0.0625 and 1 and μ isapproximately between 0 and 1. For example, α=0.25 and μ=0.0625. Theelements 1016 are element-by-element multiplied 1022 with the set ofpixels 1010 to produce elements 1024. Elements 1024 are summed (1025) toproduce value r_(0,0) 1026. Elements 1016 are also summed 1018 toproduce a time-to-scale value k_(0,0) 1020, sometimes called anormalization value. A ratio 1027 of the value r_(0,0) and thetime-to-scale value k_(0,0) 1020 is a corresponding output filteredpixel value 1028.

FIG. 11 is a flow diagram illustrating an embodiment 1100 of a processof constructing a filter kernel for a CDF and applying the CDF to aboundary pixel u_(i0). The first order difference operation is appliedto a set of pixels 1110 to produce elements 1112 that are mapped using alevel and conductivity chart (such as that shown in Table 1) to producediscrete conductivity elements 1114. Filter kernel 1116 is constructedby multiplying the predefined subsets of the discrete conductivityelements 1114 with the smoothing and sharpening parameters α and μ. Theelements 1116 are element-by-element multiplied 1122 with the set ofpixels 1110 to produce elements 1124. Elements 1124 are summed (1125) toproduce value r_(i,0) 1126. Elements 1116 are summed 1118 to producetime-to-scale value k_(i,0) 1120, sometimes called a normalizationvalue. A ratio 1127 of the value r_(i,0) and the time-to-scale valuek_(i,0) 1120 is a corresponding output filtered pixel value 1128 thatcorresponds to the input pixel value u_(i,0).

FIG. 12 is a flow diagram illustrating an embodiment 1200 of a processof constructing a filter kernel for a CDF and applying the CDF to aninterior (center) pixel u_(ij). The first order difference operation isapplied to a set of pixels 1210 to produce elements 1212 that are mappedusing a level and conductivity chart (such as that shown in Table 1) toproduce discrete conductivity elements 1214. Filter kernel 1216 isconstructed by multiplying predefined subsets of the discreteconductivity elements 1214 with the smoothing and sharpening parametersα and μ. The elements of the filter kernel 1216 are element-by-elementmultiplied 1222 with the set of pixels 1210 to produce elements 1224.Elements 1224 are summed (1225) to produce value r_(i,j) 1226. Elements1216 are summed 1218 to produce time-to-scale value k_(i,j) 1220. Theratio 1227 of the value r_(ij) 1226 and the time-to-scale value k_(i,j)1220 is computed to generate an output filtered pixel value 1228(r_(i,j)/k_(i,j)) that corresponds to the input pixel value u_(i,j).

FIG. 13 is a flow diagram illustrating an embodiment 1300 of a processof constructing and using a filter kernel for a CDF. The first-orderdifference operation is applied to a set of pixels 1310 with a centerpixel located on an image edge (within a image tile or array) to produceelements 1311. The set of pixels 1310 in this example represent avertical edge between light and dark regions of an image. Elements 1311are quantized to the corresponding conductivity level to producediscrete conductivity function values 1312, and mapped using a level andconductivity chart (such as that shown in Table 1) to produce elements1314. Predefined subsets of elements 1314 are multiplied by smoothingand sharpening parameters α and β to produce filter kernel 1316. The setof pixels 1310 are element-by-element multiplied (1322) by the filterkernel 1316 to produce elements 1324. Elements 1324 are summed (1325) toproduce value r_(i,j)=28.9316 (1326). The filter kernel elements 1316are summed 1318 to produce time-to-scale value k_(i,j)=3.0051 (1320).The ratio 1327 of the value r_(i,j) 1326 and the time-to-scale valuek_(i,j) 1320 is computed to generate an output filtered pixel value 1328(r_(i,j)/k_(i,j)=9.6275) that corresponds to the input pixel valueu_(i,j). In this example, the filtered pixel 1328 may be quantized to avalue of 10. In embodiment 1300, α=0.5, i=0.25, K=50 and theconductivity function is chosen to be

${c(p)} = {{\mathbb{e}}^{\frac{- p^{2}}{2p_{o}^{2}}}.}$

FIG. 14 is a flow diagram illustrating an embodiment 1400 of a processof constructing and using a filter kernel for a CDF. The first-orderdifference operation is applied to a set of pixels 1410 with the centerpixel not on an image edge to produce elements 1411, quantized to thecorresponding conductivity levels to produce discrete conductivityfunction values 1412, and mapped using a level and conductivity chart(such as that shown in Table 1) to produce elements 1414. Predefinedsubsets of the elements 1414 are multiplied by smoothing and sharpeningparameters α and μ and element-by-element (or term-wise) multiplied toproduce filter kernel (CDF) 1416. Input image block 1410 with initialcenter pixel=230 is now element by element multiplied 1422 by the CDF1416 to produce elements 1424. Elements 1424 are summed (1425) toproduce value r_(i,j) (921.826) 1426, and the elements of CDF 1416 aresummed 1418 to produce time-to-scale value k_(i,j) (3.995) 1420. Theratio 1427 of the value r_(ij) 1426 and the time-to-scale value k_(ij)1420 is computed to generate an output filtered pixel value (230.745)1428 that corresponds to the input pixel value u_(i,j). The filteredpixel 1428 may be quantized to a value of 231. In embodiment 1400,α=0.5, μ=0.25, K=50 and the conductivity function is chosen to be

${c(p)} = {{\mathbb{e}}^{\frac{- p^{2}}{2p_{o}^{2}}}.}$

The CDF may be used as a pre-coder for removing or reducing noise in animage, which may be an I-frame of a video sequence. For example, inembodiment 1500 in FIG. 5 an 8×8 tile 1512 of an image or I-frame isprocessed by a CDF 1514. The filtered image tile of the image or I-frame1516 is sent to encoder 1518. The encoder 1518 includes a discretecosine transformation (DCT) 1520, quantization 1522, and entropy coding1524. An encoded bit stream 1526 with a smaller bit-rate is output. Theflow diagram 1500 may include fewer operations or additional operations.In addition, two or more operations may be combined and/or the order ofthe operations may be changed.

A respective CDF may be integrated as a pre-coder in an existingencoder, such as MJPEG, MPEG-1, 2 4, 7 and H263, AVC or H264. Filteredimages generated using the respective CDF or a set of CDFs may becompatible with existing decoders. The respective filter kernel for aCDF may be used to clean up I frames, P frames, and/or macro-blocks inmotion estimation-compensation. FIG. 16 is a block diagram illustratingan embodiment 1600 of such a process. A filter kernel 1612 is applied toa motion-compensated P frame 1608 and subtracted 1614 from a filtered Iframe 1610 (which was illustrated in FIG. 15). A DCT 1618 andquantization 1620 are applied to a resulting filtered B/P frame 1616.

FIG. 17 is a block diagram illustrating an embodiment 1700 of a processof filtering 1718 a digital image or image tile, such as a frame 1712,and in particular, for filtering 1718 16×16 macro-blocks for motioncompensation. In some embodiments, a previous I or P frame 1710 is usedto determine a motion vector 1716_1 and/or a subsequent P or B frame1714 is used to determine a motion vector 1716_2. The former may occurif the frame 1712 is a P frame and the latter may occur if the frame1712 is a B frame.

FIG. 18 is a flow diagram illustrating an embodiment of a process 1800of filtering or pre-coding a three-dimensional digital image called animage cube. In process 1800, a 3-dimensional CDF is generated andapplied 1816 to reduce or eliminate noise from image cubes, such cubesbased on three consecutive video frames 1810, 1812, 1814, after motionestimation and compensation have been applied to the P-frames andB-frames (which was illustrated in FIG. 17) to create video frames 1810,1812, and 1814. In some embodiments, such as MJEP, video frames 1810,1812, and 1814 are I frames. In some embodiments, the filter kernels forCDFs may be applied to a mixture of I, P, and/or B frames thatconstitute an image cube.

The filter kernel for a CDF may be applied to one or more images, or oneor more subsets of one or more images, to reduce or eliminate noisewhile leaving image content, such as texture and/or spatial frequenciesassociated with image edges, approximately unchanged. This isillustrated in FIG. 19, which shows an unfiltered digital image and afiltered digital image generated using an embodiment of the presentinvention.

The foregoing descriptions of specific embodiments of the presentinvention are presented for purposes of illustration and description.They are not intended to be exhaustive or to limit the invention to theprecise forms disclosed. Rather, it should be appreciated that manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen and described in order to bestexplain the principles of the invention and its practical applications,to thereby enable others skilled in the art to best utilize theinvention and various embodiments with various modifications as aresuited to the particular use contemplated.

1. A method of filtering a digital image, comprising in a device havingone or more processors and memory storing one or more programs to beexecuted by the one or more processors: for each respective pixel in aset of pixels of the digital image, generating a pixel-dependent filterkernel for the respective pixel, and applying the filter kernel to therespective pixel to smooth noise and preserve spatial frequenciesassociated with image edges in the digital image in accordance with afirst filtering parameter, wherein the filter kernel is a closed-formfunction of the respective pixel, the closed-form function includingcontributions from a first set of neighboring pixels, wherein thecontribution from each respective neighboring pixel in the first set ofneighboring pixels is determined at least in part by a conductivityfunction evaluated in accordance with a difference in intensity betweenthe respective pixel and the respective neighbor pixel, the conductivityfunction mapping quantized input values to a finite number of predefinedconductivity function values, and wherein the filter kernel for eachrespective pixel in the set of pixels has a content-dependentnormalization specific to the respective pixel such that a sum of valuesof elements in the filter kernel equals a substantially fixed value. 2.The method of claim 1, wherein the applying comprises a single pass forthe respective pixel.
 3. The method of claim 1, wherein thecontent-dependent normalization of the filter kernel includes acontent-dependent scale value for the respective pixel, the methodfurther including generating the content-dependent scale value inaccordance with a time-to-scale transformation for the respective pixel.4. The method of claim 1, wherein applying the filter kernel to eachrespective pixel in the set of pixels leaves the filtered digital imagesubstantially unchanged.
 5. The method of claim 1, wherein the firstfiltering parameter determines a boundary between smoothing andsharpening.
 6. The method of claim 1, wherein the first filteringparameter substantially maintains texture information in the digitalimage by reducing smoothing of noise at a position in the digital imagecontaining spatial frequencies greater than a threshold.
 7. The methodof claim 1, wherein non-central elements in the filter kernel for therespective pixel comprise values of a discretization of an anisotropicdiffusion equation determined based on magnitudes of differences betweenpixels in the first set of neighboring pixels and the respective pixel,a central pixel in the filter kernel corresponds to the respectivepixel, and wherein the first filtering parameter maps a time step in thediscretization of the anisotropic diffusion equation to acontent-dependent scale.
 8. The method of claim 7, wherein aconductivity in the discretization of the anisotropic diffusion equationis a function of a wavelet transformation.
 9. The method of claim 8,wherein the wavelet transformation is implemented as a look-up table.10. The method of claim 1, further comprising modifying a color of thepixel using the filter kernel.
 11. The method of claim 10, whereinmodifying the color includes a color component.
 12. The method of claim1, wherein the filter kernel includes contributions from a second set ofneighboring pixels and is farther in accordance with a second filteringparameter.
 13. The method of claim 1, wherein the filter kernelcorresponds to a window of size 2m+1 by 2n+1 and includes the first setof neighboring pixels.
 14. The method of claim 13, wherein m is equal ton.
 15. The method of claim 13, wherein m and n for the respective pixelare selected from a set consisting of positive integers between 1 and50.
 16. The method of claim 13, wherein m and n for the filter kernelare a function of a pixel size and image resolution.
 17. The method ofclaim 1, wherein the digital image comprises video data.
 18. The methodof claim 1, wherein the applying is performed for each pixel in the setof pixels to produce a pre-coded digital image, the method furthercomprising compressing the pre-coded digital image in accordance with apredefined image compression methodology.
 19. A method of filtering adigital image, comprising: in a device having one or more processors andmemory storing one or more programs to be executed by the one or moreprocessors: applying a respective pixel-dependent filter kernel to eachrespective pixel in a set of pixels to smooth noise and preserve spatialfrequencies associated with image edges in the digital image inaccordance with a filtering parameter to produce a filtered pixel,wherein the filter kernel is a closed-form function of the respectivepixel, the closed-form function including contributions from a first setof neighboring pixels, wherein the contribution from each respectiveneighboring pixel in the first set of neighboring pixels is determinedat least in part by a conductivity function evaluated in accordance witha difference in intensity between the respective pixel and therespective neighbor pixel, the conductivity function mapping quantizedinput values to a finite number of predefined conductivity functionvalues, and wherein the filtered pixel corresponds to${\sum\limits_{m,n}\;{\frac{1}{\gamma}{M \otimes U}}},$ M is aclosed-form array that is a function of the respective pixel and has awindow size of 2m+1 by 2n+1, where m and n are positive integers, andwherein M includes contributions from a set of neighboring pixels in thewindow, U is a sub-array in the set of pixels and includes therespective pixel and the set of neighboring pixels, {circle around (×)}indicates element-by-element multiplication of elements in M andelements in U, and γ is a content-dependent normalization factorspecific to the respective pixel such that a sum of value of elements inthe array $\frac{1}{\gamma}M$ equals a substantially fixed value.
 20. Adigital image processor, comprising: memory; a processor; and one ormore programs stored in the memory and configured to be executed by theprocessor, the one or more programs including: instructions forgenerating, for each respective pixel in a set of pixels of the digitalimage, a pixel-dependent filter kernel for the respective pixel, and forapplying the filter kernel to the respective pixel to smooth noise andpreserve spatial frequencies associated with image edges in the digitalimage in accordance with a filtering parameter, wherein the filterkernel is a closed-form function of the respective pixel, theclosed-form function including contributions from a first set ofneighboring pixels, wherein the contribution from each respectiveneighboring pixel in the first set of neighboring pixels is determinedat least in part by a conductivity function evaluated in accordance witha difference in intensity between the respective pixel and therespective neighbor pixel, the conductivity function mapping quantizedinput values to a finite number of predefined conductivity functionvalues, and wherein the filter kernel for each respective pixel in theset of pixels has a content-dependent normalization specific to therespective pixel such that a sum of values of elements in the filterkernel equals a substantially fixed value.
 21. A computer-programproduct for filtering a digital image, comprising: a computer readablestorage medium and a computer program mechanism embedded therein, thecomputer program mechanism comprising: instructions for generating, foreach respective pixel in a set of pixels of the digital image, apixel-dependent filter kernel for the respective pixel, and for applyingthe filter kernel to the respective pixel to smooth noise and preservespatial frequencies associated with image edges in the digital image inaccordance with a filtering parameter, wherein the filter kernel is aclosed-form function of the respective pixel the closed-form functionincluding contributions from a first set of neighboring pixels, whereinthe contribution from each respective neighboring pixel in the first setof neighboring pixels is determined at least in part by a conductivityfunction evaluated in accordance with a difference in intensity betweenthe respective pixel and the respective neighbor pixel, the conductivityfunction mapping quantized input values to a finite number of predefinedconductivity function values, and wherein the filter kernel for eachrespective pixel in the set of pixels has a content-dependentnormalization specific to the respective pixel such that a sum of valuesof elements in the filter kernel equals a substantially fixed value. 22.A digital image processor, comprising: memory means; processor means;and a program mechanism, wherein the program mechanism is stored in thememory means and configured to be executed by the processor means, theprogram mechanism including: instructions for generating, for eachrespective pixel in a set of pixels of the digital image, apixel-dependent filter kernel for the respective pixel, and for applyingthe filter kernel to the respective pixel to smooth noise and preservespatial frequencies associated with image edges in the digital image inaccordance with a filtering parameter, wherein the filter kernel is aclosed-form function of the respective pixel the closed-form functionincluding contributions from a first set of neighboring pixels, whereinthe contribution from each respective neighboring pixel in the first setof neighboring pixels is determined at least in part by a conductivityfunction evaluated in accordance with a difference in intensity betweenthe respective pixel and the respective neighbor pixel, the conductivityfunction mapping quantized input values to a finite number of predefinedconductivity function values, and wherein the filter kernel for eachrespective pixel in the set of pixels has a content-dependentnormalization specific to the respective pixel such that a sum of valuesof elements in the filter kernel equals a substantially fixed value. 23.The digital image processor of claim 20, wherein the content-dependentnormalization of the filter kernel includes a content-dependent scalevalue for the respective pixel, and the one or more programs furtherincluding instructions for generating the content-dependent scale valuein accordance with a time-to-scale transformation for the respectivepixel.
 24. The computer program product of claim 21, wherein thecontent-dependent normalization of the filter kernel includes acontent-dependent scale value for the respective pixel, and the computerprogram mechanism further comprising instructions for generating thecontent-dependent scale value in accordance with a time-to-scaletransformation for the respective pixel.
 25. The method of claim 1,wherein the respective pixel is in a respective video frame of asequence of video frames, the set of neighboring pixels includes: one ormore corresponding pixels, corresponding to the respective pixel, in avideo frame before the respective video frame in the sequence, and oneor more corresponding pixels, corresponding to the respective pixel, ina video frame after the respective video frame; and the conductivityfunction is a three dimensional conductivity function.
 26. The digitalimage processor of claim 20, wherein the respective pixel is in arespective video frame of a sequence of video frames, the set ofneighboring pixels includes: one or more corresponding pixels,corresponding to the respective pixel, in a video frame before therespective video frame in the sequence, and one or more correspondingpixels, corresponding to the respective pixel, in a video frame afterthe respective video frame; and the conductivity function is a threedimensional conductivity function.
 27. The computer program product ofclaim 21, wherein the respective pixel is in a respective video frame ofa sequence of video frames, the set of neighboring pixels includes: oneor more corresponding pixels, corresponding to the respective pixel, ina video frame before the respective video frame in the sequence, and oneor more corresponding pixels, corresponding to the respective pixel, ina video frame after the respective video frame; and the conductivityfunction is a three dimensional conductivity function.